Failure Analysis in Backtrack Search for Constraint Satisfaction

Researchers have focused on assessing the quality of search algorithms by measuring effort, number of mistakes, runtime distributions and other characteristics. In this paper we attempt to enhance our understanding of these algorithms by employing heatmap-based visualisations. We perform a large empirical study of search, for a variety of problem classes, and we report several interesting observations. Our results show that, contrary to conventional wisdom, it is not always the case that mistakes made at the top of the search tree are exponentially more expensive to refute than those made deeper in the tree, or that over a population of instances the largest mistakes dominate search effort. Furthermore, we show empirically that heavy-tailed runtime distributions can occur even if the number of mistakes is small, and that instance and mistake-based runtime distributions can be very similar. Finally, we study the complex relationship between the promise and fail-firstness of variable ordering heuristics.